Audio-driven talking face generation has garnered significant interest within the domain of digital human research. Existing methods are encumbered by intricate model architectures that are intricately dependent on each other, complicating the process of re-editing image or video inputs. In this work, we present ControlTalk, a talking face generation method to control face expression deformation based on driven audio, which can construct the head pose and facial expression including lip motion for both single image or sequential video inputs in a unified manner. By utilizing a pre-trained video synthesis renderer and proposing the lightweight adaptation, ControlTalk achieves precise and naturalistic lip synchronization while enabling quantitative control over mouth opening shape. Our experiments show that our method is superior to state-of-the-art performance on widely used benchmarks, including HDTF and MEAD. The parameterized adaptation demonstrates remarkable generalization capabilities, effectively handling expression deformation across same-ID and cross-ID scenarios, and extending its utility to out-of-domain portraits, regardless of languages.

One of the most challenging problems in audio-driven talking head generation is achieving high-fidelity detail while ensuring precise synchronization. Given only a single reference image, extracting meaningful identity attributes becomes even more challenging, often causing the network to mirror the facial and lip structures too closely. To address these issues, we introduce RADIO, a framework engineered to yield high-quality dubbed videos regardless of the pose or expression in reference images. The key is to modulate the decoder layers using latent space composed of audio and reference features. Additionally, we incorporate ViT blocks into the decoder to emphasize high-fidelity details, especially in the lip region. Our experimental results demonstrate that RADIO displays high synchronization without the loss of fidelity. Especially in harsh scenarios where the reference frame deviates significantly from the ground truth, our method outperforms state-of-the-art methods, highlighting its robustness.

The Lip Synch challenge, recently introduced by Google's AI Experiment group, aims at teaching the tech giant's AI system the art of reading lips. This initiative is being executed to help Google develop applications for people with speaking disabilities, such as Amyotrophic lateral sclerosis (ALS). Google plans to take assistance from professional singers to help their AI systems learn the skill of synchronisation. The platform is very self-descriptively named Lip Synch and is built by YouTube for Chrome on desktop. Lip Sync offers participants to sing a particular segment of the "Dance Monkey" by Tones and I, the only permissible sound bite accepted currently.

Lip-reading is notoriously difficult, depending as much on context and knowledge of language as it does on visual clues. But researchers are showing that machine learning can be used to discern speech from silent video clips more effectively than professional lip-readers can. In one project, a team from the University of Oxford's Department of Computer Science has developed a new artificial-intelligence system called LipNet. As Quartz reported, its system was built on a data set known as GRID, which is made up of well-lit, face-forward clips of people reading three-second sentences. Each sentence is based on a string of words that follow the same pattern.

Google engineer Supasorn Suwajanakorn developed a tool which, fed with the right input, can create a realistic fake video that mimics the way a person talks by closely observing existing footage of their mouth and teeth to create the perfect lip-sync. It could be used to create videos of dead relatives - but also to create'deepfake' videos for nefarious purposes, he warned. Such technology could be used to create virtual versions of those who have passed - grandparents could be asked for advice; actors returned to the screen; great teachers give lessons, or authors read their works aloud, according to Suwajanakorn. 'Wouldn't it be great if you could ask our grandparents for advice and hear those comforting words, even if they're no longer with us?' he told the TED Conference in Vancouver earlier this year. However, he also revealed he has also developed a'Reality Defender' app to spot the deepfake videos created using the technology.

It's terrifyingly easy to just make stuff up online these days, such is life in the post-truth era. But recent advancements in machine learning (ML) and artificial intelligence (AI) have compounded the issue exponentially. It's not just the news that's fake anymore but all sorts of media and consumer goods can now be knocked off thanks to AI. From audio tracks and video clips to financial transactions and counterfeit products -- even your own handwriting can be mimicked with startling levels of accuracy. But what if we could leverage the same computer systems that created these fakes to reveal them just as easily? People have been falling for trickery and hoaxes since forever.

This might come across surprising for lot many AI enthusiasts out there, but the technology actually fosters the capability to create fake audio and video, which is difficult to distinguish from reality. In a recent feat, scientists at University of Washington created an AI software that could generate highly realistic fake videos of former president Barack Obama using existing audio and video clips of him. The tool essentially takes audio files, converts them into realistic mouth movements, and then grafts those movements onto existing video. The resultant video shows someone saying something they didn't. University of Washington scientists had previously revealed that the tool could be utilized for generating digital doppelgangers of anyone by simply analyzing their images.

University of Washington researchers at the UW Graphics and Image Laboratory have developed new algorithms that turn audio clips into a realistic, lip-synced video, starting with an existing video of that person speaking on a different topic. As detailed in a paper to be presented Aug. 2 at SIGGRAPH 2017, the team successfully generated a highly realistic video of former president Barack Obama talking about terrorism, fatherhood, job creation and other topics, using audio clips of those speeches and existing weekly video addresses in which he originally spoke on a different topic decades ago. Realistic audio-to-video conversion has practical applications like improving video conferencing for meetings (streaming audio over the internet takes up far less bandwidth than video, reducing video glitches), or holding a conversation with a historical figure in virtual reality, said Ira Kemelmacher-Shlizerman, an assistant professor at the UW's Paul G. Allen School of Computer Science & Engineering. This beats previous audio-to-video conversion processes, which have involved filming multiple people in a studio saying the same sentences over and over to try to capture how a particular sound correlates to different mouth shapes, which is expensive, tedious and time-consuming. The new machine learning tool may also help overcome the "uncanny valley" problem, which has dogged efforts to create realistic video from audio.

The technology is based on newly prepared algorithms, which are designed to overcome a limitation with'computer vision'. This is with turning audio clips into realistic, lip-synced videos of the person who is speaking the words. The developed algorithms learn from videos that exist "in the wild", such as on the Internet or elsewhere. READ MORE: New software edits voices like text To do so involved training a neural network (a collection of connected units called artificial neurons) to view videos of an individual and then to translate different audio sounds into basic mouth shapes. The second area was using a new mouth synthesis technique to realistically superimpose mouth shapes and textures onto an existing reference video of a given person.

Video Researchers have crafted algorithms that can take an audio recording of someone talking and map it to a video clip of them speaking to create a new convincing lip-synched video with the replacement sound. In other words, the resulting video carries the injected audio, rather than its original sound, and the frames are manipulated so that the speaker's face and mouth movements match the new audio. You can be forgiven for seeing this as a vital stepping stone to creating the ultimate fake news – highly believable forged videos. Imagine taking a clip of someone important speaking at a private event, and using the aforementioned software to dub in a completely new script, voiced by a skilled impersonator or generated by another AI such as Lyrebird, and then distributing that fraudulent footage. Thankfully, technology is nowhere near that level right now.